feat: add AWS Bedrock function calling implementation #120
Conversation
- Implemented function calling for AWS Bedrock Realtime (Nova Sonic) - Added tool schema conversion to AWS Nova format - Implemented tool execution handlers for realtime API - Added audio resampling fix for simple_audio_response - Created example demonstrating function calling with AWS LLM - Updated README with function calling documentation - Added test for AWS Realtime function calling Note: AWS Nova Realtime toolConfiguration causes connection errors, likely an API limitation. Implementation is ready for when AWS adds support.
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughAdds AWS Bedrock function-calling to Standard LLM and Realtime flows, updates README/examples to use cartesia.TTS, introduces Nova-compatible raw event templates and tool lifecycle methods, adds examples and an integration test, and hardens logging, error handling, and tool conversion across streaming/non-streaming paths. Changes
Sequence Diagram(s)sequenceDiagram
participant User as User/Agent
participant Realtime as Realtime (plugin)
participant Bedrock as AWS Bedrock (Nova)
participant Tool as Registered Tool
User->>Realtime: send prompt requesting tool use
Realtime->>Bedrock: send text input + tools config (TEXT_INPUT_EVENT)
Bedrock-->>Realtime: emit toolUse event (tool name, args, tool_use_id)
Realtime->>Realtime: enqueue pending tool call (by id)
Realtime->>Tool: execute registered function (async)
Tool-->>Realtime: return result / error
Realtime->>Bedrock: send toolResult (send_tool_result) + content_end
Bedrock-->>Realtime: emit text/audio response
Realtime->>User: emit LLMResponseCompletedEvent (final text/audio)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Areas requiring extra attention:
Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
commit d78a4a0 Author: Dan Gusev <[email protected]> Date: Thu Oct 30 20:44:19 2025 +0100 Logging cleanup (#133) * logging: use logging.getLogger(__name__) everywhere to simplify configuration * Clean up logging everywhere - Replaced "logging.info" usages with separate loggers - Lowered some info messages to debug - Replaced prints with logging - Added emojis where they're already used * Enable default logging for the SDK logs - Added a way to set the SDK log level at the Agent class - Set the default formatter and added level-based coloring - If the logs are already configured, they remain intact - Moved logging_utils.py to utils/logging.py * Remove "name" from the default logging formatter --------- Co-authored-by: Thierry Schellenbach <[email protected]>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR is being reviewed by Cursor Bugbot
Details
Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
agents-core/vision_agents/core/agents/agents.py (1)
534-548: Remove duplicate connection check (dead code).Lines 534-539 check
if not self._connectionand return early. Lines 544-548 perform an identical check inside the span, which is unreachable dead code. Remove the duplicate check at lines 544-548.Apply this diff:
# If connection is None or already closed, return immediately if not self._connection: self.logger.info( "🔚 Agent connection is already closed, finishing immediately" ) return - - with self.span("agent.finish"): - # If connection is None or already closed, return immediately - if not self._connection: - logging.info( - "🔚 Agent connection already closed, finishing immediately" - ) - return + with self.span("agent.finish"): @self.edge.events.subscribe async def on_ended(event: CallEndedEvent): self._is_running = False
♻️ Duplicate comments (1)
plugins/aws/vision_agents/plugins/aws/aws_realtime.py (1)
408-418: Fix JSON escaping intext_input.Manual string interpolation still misses characters like backslashes and tabs, so any such input breaks the JSON sent to Nova. Please build the payload as a dict and let
json.dumpshandle escaping.Apply this diff:
- # Escape content for JSON - escaped_content = ( - content.replace('"', '\\"').replace("\n", "\\n").replace("\r", "\\r") - ) - event_json = self.TEXT_INPUT_EVENT % ( - self.session_id, - content_name, - escaped_content, - ) - await self.send_raw_event(event_json) + event = { + "event": { + "textInput": { + "promptName": self.session_id, + "contentName": content_name, + "content": content, + } + } + } + await self.send_event(event)
🧹 Nitpick comments (4)
plugins/getstream/vision_agents/plugins/getstream/stream_edge_transport.py (1)
411-413: Add an explanatory comment for the demo video bitrate choice.The bitrate of 12000000 (12 Mbps) for 1080p is higher than typical streaming standards (4-8 Mbps). Since the H.264 codec choice already has a comment explaining compatibility reasoning, consider adding a similar comment to lines 411-413 explaining why this higher bitrate was chosen for the demo experience—whether it's for visual fidelity, specific quality requirements, or testing purposes.
agents-core/vision_agents/core/mcp/mcp_base.py (2)
77-79: Add type hint foroperation_funcparameter.The
operation_funcparameter lacks a type annotation. Consider addingCallabletype hint for better type safety.Apply this diff:
+from typing import Callable + - async def _call_with_retry( - self, operation_name: str, operation_func, *args, **kwargs - ): + async def _call_with_retry( + self, operation_name: str, operation_func: Callable, *args, **kwargs + ) -> Any:
84-86: Consider removing redundant_update_activity()calls.Activity is updated both here (line 85) and within each
_*_implmethod (lines 105, 116, 124, 133, 143, 154). While not harmful, this duplication is unnecessary. Consider removing the_update_activity()calls from the_*_implmethods since they're already called in_call_with_retry.agents-core/vision_agents/core/utils/logging.py (1)
40-47: Consider usingaddHandler()instead of direct list assignment.While functionally correct, line 43's direct assignment to
_logger.handlers = [default_handler]is less idiomatic than_logger.addHandler(default_handler). The standard approach is more explicit and integrates better with Python's logging internals.Apply this diff:
for _logger in [MAIN_LOGGER, GETSTREAM_LOGGER]: # Set the default handler only if it's not already configured if not _logger.handlers: - _logger.handlers = [default_handler] + _logger.addHandler(default_handler) _logger.propagate = False # Do not override the level if it's set if _logger.level == logging.NOTSET: _logger.setLevel(level)
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (3)
examples/01_simple_agent_example/uv.lockis excluded by!**/*.lockplugins/aws/example/uv.lockis excluded by!**/*.lockuv.lockis excluded by!**/*.lock
📒 Files selected for processing (20)
agents-core/pyproject.toml(1 hunks)agents-core/vision_agents/core/agents/agents.py(18 hunks)agents-core/vision_agents/core/events/manager.py(5 hunks)agents-core/vision_agents/core/mcp/mcp_base.py(2 hunks)agents-core/vision_agents/core/mcp/mcp_manager.py(3 hunks)agents-core/vision_agents/core/processors/base_processor.py(5 hunks)agents-core/vision_agents/core/utils/logging.py(2 hunks)plugins/aws/README.md(4 hunks)plugins/aws/example/aws_llm_function_calling_example.py(1 hunks)plugins/aws/example/aws_realtime_function_calling_example.py(1 hunks)plugins/aws/example/pyproject.toml(1 hunks)plugins/aws/tests/test_aws_realtime.py(1 hunks)plugins/aws/vision_agents/plugins/aws/aws_llm.py(15 hunks)plugins/aws/vision_agents/plugins/aws/aws_realtime.py(15 hunks)plugins/cartesia/vision_agents/plugins/cartesia/tts.py(2 hunks)plugins/elevenlabs/vision_agents/plugins/elevenlabs/tts.py(2 hunks)plugins/getstream/vision_agents/plugins/getstream/stream_edge_transport.py(8 hunks)plugins/kokoro/vision_agents/plugins/kokoro/tts.py(2 hunks)plugins/krisp/vision_agents/plugins/krisp/turn_detection.py(1 hunks)tests/test_utils.py(2 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (.cursor/rules/python.mdc)
**/*.py: Do not modify sys.path in Python code
Docstrings must follow the Google style guide
Files:
plugins/elevenlabs/vision_agents/plugins/elevenlabs/tts.pyagents-core/vision_agents/core/processors/base_processor.pyagents-core/vision_agents/core/events/manager.pyplugins/krisp/vision_agents/plugins/krisp/turn_detection.pyplugins/cartesia/vision_agents/plugins/cartesia/tts.pyagents-core/vision_agents/core/utils/logging.pytests/test_utils.pyplugins/aws/vision_agents/plugins/aws/aws_llm.pyplugins/aws/tests/test_aws_realtime.pyagents-core/vision_agents/core/mcp/mcp_base.pyplugins/aws/example/aws_realtime_function_calling_example.pyplugins/kokoro/vision_agents/plugins/kokoro/tts.pyplugins/aws/example/aws_llm_function_calling_example.pyplugins/getstream/vision_agents/plugins/getstream/stream_edge_transport.pyagents-core/vision_agents/core/agents/agents.pyagents-core/vision_agents/core/mcp/mcp_manager.pyplugins/aws/vision_agents/plugins/aws/aws_realtime.py
tests/**/*.py
📄 CodeRabbit inference engine (.cursor/rules/python.mdc)
tests/**/*.py: Never use mocking utilities (e.g., unittest.mock, pytest-mock) in test files
Write tests using pytest (avoid unittest.TestCase or other frameworks)
Mark integration tests with @pytest.mark.integration
Do not use @pytest.mark.asyncio; async support is automatic
Files:
tests/test_utils.py
🧠 Learnings (1)
📚 Learning: 2025-10-13T22:00:34.300Z
Learnt from: dangusev
Repo: GetStream/Vision-Agents PR: 98
File: plugins/deepgram/vision_agents/plugins/deepgram/stt.py:135-150
Timestamp: 2025-10-13T22:00:34.300Z
Learning: In the Deepgram STT plugin (plugins/deepgram/vision_agents/plugins/deepgram/stt.py), the `started()` method is designed to wait for the connection attempt to complete, not to guarantee a successful connection. It's acceptable for the connection attempt to fail, and downstream code handles the case where `self.dg_connection` is `None`. The `_connected_once` event is set in the `finally` block intentionally to signal attempt completion.
Applied to files:
agents-core/vision_agents/core/agents/agents.py
🧬 Code graph analysis (9)
tests/test_utils.py (2)
agents-core/vision_agents/core/utils/video_utils.py (1)
ensure_even_dimensions(6-26)agents-core/vision_agents/core/utils/logging.py (1)
configure_default_logging(20-47)
plugins/aws/vision_agents/plugins/aws/aws_llm.py (5)
agents-core/vision_agents/core/llm/events.py (2)
LLMResponseChunkEvent(90-105)LLMResponseCompletedEvent(109-115)plugins/aws/vision_agents/plugins/aws/aws_realtime.py (1)
_convert_tools_to_provider_format(448-500)agents-core/vision_agents/core/llm/llm.py (2)
LLMResponseEvent(25-29)_dedup_and_execute(318-350)agents-core/vision_agents/core/llm/llm_types.py (2)
NormalizedToolCallItem(107-111)ToolSchema(64-67)plugins/aws/vision_agents/plugins/aws/events.py (1)
AWSStreamEvent(7-10)
plugins/aws/tests/test_aws_realtime.py (3)
agents-core/vision_agents/core/llm/llm.py (1)
register_function(170-183)agents-core/vision_agents/core/llm/events.py (1)
RealtimeAudioOutputEvent(37-45)plugins/aws/vision_agents/plugins/aws/aws_realtime.py (1)
simple_response(266-281)
agents-core/vision_agents/core/mcp/mcp_base.py (1)
agents-core/vision_agents/core/mcp/mcp_manager.py (1)
call_tool(93-111)
plugins/aws/example/aws_realtime_function_calling_example.py (5)
agents-core/vision_agents/core/edge/types.py (1)
User(22-25)agents-core/vision_agents/core/agents/agents.py (2)
Agent(107-1327)finish(529-560)plugins/aws/example/aws_llm_function_calling_example.py (3)
start_agent(17-84)get_weather(37-42)calculate(48-55)plugins/aws/vision_agents/plugins/aws/aws_realtime.py (1)
simple_response(266-281)agents-core/vision_agents/core/llm/llm.py (1)
register_function(170-183)
plugins/aws/example/aws_llm_function_calling_example.py (4)
agents-core/vision_agents/core/agents/agents.py (1)
Agent(107-1327)plugins/aws/example/aws_realtime_function_calling_example.py (2)
get_weather(46-69)calculate(75-105)agents-core/vision_agents/core/llm/llm.py (2)
LLM(36-363)register_function(170-183)plugins/aws/vision_agents/plugins/aws/aws_llm.py (1)
simple_response(82-102)
agents-core/vision_agents/core/agents/agents.py (1)
agents-core/vision_agents/core/utils/logging.py (4)
CallContextToken(51-55)clear_call_context(97-104)set_call_context(84-94)configure_default_logging(20-47)
agents-core/vision_agents/core/mcp/mcp_manager.py (1)
agents-core/vision_agents/core/mcp/mcp_base.py (1)
MCPBaseServer(10-189)
plugins/aws/vision_agents/plugins/aws/aws_realtime.py (3)
agents-core/vision_agents/core/llm/realtime.py (2)
Realtime(24-203)_emit_audio_output_event(119-131)agents-core/vision_agents/core/llm/llm.py (3)
_build_enhanced_instructions(61-85)get_available_functions(185-187)_run_one_tool(231-297)plugins/aws/vision_agents/plugins/aws/aws_llm.py (1)
_convert_tools_to_provider_format(625-674)
🪛 GitHub Actions: CI (unit)
plugins/aws/example/aws_realtime_function_calling_example.py
[error] 3-3: F401 'pathlib.Path' imported but unused. Remove unused import: pathlib.Path. 2 fixable with the --fix option.
plugins/aws/example/aws_llm_function_calling_example.py
[error] 9-9: F401 import 'vision_agents.plugins.smart_turn' imported but unused. Remove unused import: vision_agents.plugins.smart_turn. 2 fixable with the --fix option.
🔇 Additional comments (24)
plugins/elevenlabs/vision_agents/plugins/elevenlabs/tts.py (2)
9-10: LGTM: Module-level logger follows best practices.Using
logging.getLogger(__name__)creates a properly scoped logger for this module, which is the recommended approach for Python logging.
74-74: LGTM: Logging call updated consistently.The switch from
logging.infotologger.infocorrectly uses the module-level logger declared above, completing the logging refactor for this file.plugins/cartesia/vision_agents/plugins/cartesia/tts.py (2)
17-18: LGTM! Module-level logger follows best practices.This addition establishes a proper module-level logger using the standard Python logging pattern, enabling better log routing and configuration through the logger hierarchy.
94-94: LGTM! Consistent usage of module-level logger.The switch from the root logger to the module-level logger properly routes this log message through the module's logger instance, improving logging consistency.
plugins/krisp/vision_agents/plugins/krisp/turn_detection.py (1)
45-45: LGTM! Module-level logger follows Python best practices.Using
__name__instead of a hard-coded string aligns with the logging refactor and makes the logger hierarchy more maintainable. The logger will now use the module's full path, providing better context in log output.plugins/kokoro/vision_agents/plugins/kokoro/tts.py (1)
18-75: Module logger adoption looks solid.Moving to a module-scoped logger keeps this plugin consistent with the broader logging refactor and eliminates the direct
loggingcalls without altering behavior. Nicely done.agents-core/vision_agents/core/processors/base_processor.py (1)
22-199: Consistent logger usage across processors.The refactor keeps logging calls uniform and leverages the module logger without touching processing flow. Looks great.
agents-core/pyproject.toml (1)
29-30: Dependency addition aligns with logging refactor.Including
colorloghere ensures the new formatting utilities have their runtime dependency. No issues spotted.agents-core/vision_agents/core/mcp/mcp_manager.py (1)
12-28: LoggerAdapter support looks correct.Accepting either a
LoggerorLoggerAdaptermatches the new logging patterns without disturbing behavior. All good here.agents-core/vision_agents/core/events/manager.py (1)
18-512: Debug-level adjustments make sense.Dialing these informational logs down to debug should trim routine noise while keeping exception paths intact. Change looks tidy.
agents-core/vision_agents/core/mcp/mcp_base.py (2)
20-20: LGTM! Module-level logger follows best practices.The use of
logging.getLogger(__name__)aligns with the new centralized logging approach and ensures proper logger hierarchy.
157-189: LGTM! Consistent auto-reconnect pattern across all operations.The delegation to
_call_with_retryprovides uniform retry and reconnection semantics for all MCP operations, improving reliability.agents-core/vision_agents/core/utils/logging.py (1)
1-1: LGTM! Colorlog integration enhances developer experience.The new colorized logging infrastructure with dedicated loggers for "vision_agents" and "getstream" provides better observability.
Also applies to: 6-6, 16-17
tests/test_utils.py (3)
1-1: LGTM! New imports support logging tests.The added imports are properly used in the new test fixtures and test cases below.
Also applies to: 7-7, 13-13
553-573: LGTM! Well-designed fixture with proper state management.The
make_loggerfixture correctly captures and restores logger configuration, enabling isolated test execution without side effects.
575-611: LGTM! Comprehensive test coverage for logging configuration.The three tests effectively verify the key behaviors of
configure_default_logging: setting defaults, preserving existing handlers, and respecting pre-configured levels. As per coding guidelines.agents-core/vision_agents/core/agents/agents.py (6)
42-47: LGTM! Logging infrastructure imports support enhanced observability.The newly imported utilities enable call context tracking and configurable logging throughout the agent lifecycle.
78-86: LGTM! Logger adapter enriches logs with agent context.The
_AgentLoggerAdaptercorrectly implements the LoggerAdapter pattern to include agent_id in all log messages, improving traceability in multi-agent scenarios.
155-177: Verify agent ID initialization for logging consistency.Line 177 initializes the logger adapter with
self.agent_user.id, which may beNoneuntilcreate_user()is called (lines 660-661). This means early logs during initialization might display[Agent: None], which could be confusing. Consider generating the agent ID earlier or documenting this behavior.
272-280: LGTM! Proper use of instance logger and tracer.Using
self.loggerandself.tracerenables proper agent-specific logging context and custom tracing instrumentation.
213-213: LGTM! Logging improvements enhance observability.The logging changes appropriately use
self.logger(which includes agent context), adjust log levels for noise reduction (e.g., debug for VAD events), and improve message formatting for readability.Also applies to: 668-668, 714-714, 801-803, 844-846
887-912: LGTM! Formatting improvements enhance code readability.The multiline formatting of conditionals, method calls, and tuple unpacking makes the code more maintainable without changing behavior.
Also applies to: 918-965, 1039-1087
plugins/aws/example/pyproject.toml (2)
20-22: Editable install paths are correctly aligned with the PR structure.The relative paths for vision-agents-plugins-getstream, vision-agents-plugins-aws, and vision-agents use editable installs that reference the expected sibling and parent directories. This aligns with the stated PR focus on AWS Bedrock function calling and plugin integration.
2-12: Dependency updates correctly reflect the AWS Bedrock migration.The removal of Gemini-related dependencies (google-genai, gemini plugin) and addition of AWS-specific dependencies (vision-agents-plugins-aws, boto3) are well-aligned with the PR objectives. The project name update to aws-bedrock-realtime-example clearly communicates the shift in focus.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
plugins/aws/example/aws_llm_function_calling_example.py (1)
48-55: Replace eval with safe arithmetic (RCE risk remains unfixed).The
eval(expression)call is a remote-code-execution vulnerability that persists despite the prior critical review. Even example code should never demonstrate eval on untrusted input; users will copy this pattern into production. Please adopt the explicit operator approach shown inaws_realtime_function_calling_example.pylines 74-104, which safely handles add, subtract, multiply, and divide operations.Apply this diff to match the safe pattern from the realtime example:
@agent.llm.register_function( name="calculate", - description="Performs a mathematical calculation" + description="Perform a mathematical calculation" ) - def calculate(expression: str) -> dict: - """Performs a mathematical calculation.""" - logger.info(f"Tool: calculate called with expression: {expression}") + def calculate(operation: str, a: float, b: float) -> dict: + """Perform a calculation. + + Args: + operation: The operation to perform (add, subtract, multiply, divide) + a: First number + b: Second number + + Returns: + Result of the calculation + """ + logger.info(f"Tool: calculate called with operation: {operation}, a: {a}, b: {b}") + + operations = { + "add": lambda x, y: x + y, + "subtract": lambda x, y: x - y, + "multiply": lambda x, y: x * y, + "divide": lambda x, y: x / y if y != 0 else None, + } + + if operation not in operations: + return {"error": f"Unknown operation: {operation}"} + + result = operations[operation](a, b) + if result is None: + return {"error": "Cannot divide by zero"} + - try: - result = eval(expression) # DANGER: In a real app, use a safer math evaluator! - return {"expression": expression, "result": result} - except Exception as e: - return {"expression": expression, "error": str(e)} + return { + "operation": operation, + "a": a, + "b": b, + "result": result + }
🧹 Nitpick comments (4)
plugins/aws/example/aws_realtime_function_calling_example.py (1)
114-121: Hardcoded sleep durations may be fragile.While acceptable for a demonstration, the fixed 5s and 15s waits assume specific processing times. In production, consider event-driven synchronization or polling for function-call completion rather than arbitrary delays.
plugins/aws/example/aws_llm_function_calling_example.py (3)
29-29: Remove or document the commented turn_detection.The commented line suggests incomplete cleanup. Either remove it entirely if turn detection isn't needed for this example, or add a comment explaining why it's disabled.
37-42: Inconsistent temperature type in return values.Boulder returns
temperature: 72(int) while other cities returntemperature: "unknown"(str). For consistency and to avoid type-related issues, consider using a numeric sentinel (e.g.,Noneor-999) or returning a consistent string format.
62-84: Hardcoded sleep durations assume fixed processing times.The multiple 5-second delays work for demonstration but are fragile. In real usage, consider awaiting completion events or implementing a polling mechanism to detect when function calls finish, rather than guessing at timing.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (1)
plugins/aws/example/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (2)
plugins/aws/example/aws_llm_function_calling_example.py(1 hunks)plugins/aws/example/aws_realtime_function_calling_example.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (.cursor/rules/python.mdc)
**/*.py: Do not modify sys.path in Python code
Docstrings must follow the Google style guide
Files:
plugins/aws/example/aws_realtime_function_calling_example.pyplugins/aws/example/aws_llm_function_calling_example.py
🧬 Code graph analysis (2)
plugins/aws/example/aws_realtime_function_calling_example.py (4)
agents-core/vision_agents/core/edge/types.py (1)
User(22-25)agents-core/vision_agents/core/agents/agents.py (2)
Agent(107-1327)finish(529-560)plugins/aws/vision_agents/plugins/aws/aws_realtime.py (2)
Realtime(38-817)simple_response(266-281)agents-core/vision_agents/core/llm/llm.py (1)
register_function(170-183)
plugins/aws/example/aws_llm_function_calling_example.py (5)
agents-core/vision_agents/core/edge/types.py (1)
User(22-25)agents-core/vision_agents/core/agents/agents.py (2)
Agent(107-1327)finish(529-560)plugins/aws/example/aws_realtime_function_calling_example.py (3)
start_agent(20-123)get_weather(45-68)calculate(74-104)agents-core/vision_agents/core/llm/llm.py (2)
LLM(36-363)register_function(170-183)plugins/aws/vision_agents/plugins/aws/aws_llm.py (1)
simple_response(82-102)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: unit / Ruff & mypy
- GitHub Check: unit / Test "not integration"
- GitHub Check: unit / Test "not integration"
- GitHub Check: unit / Ruff & mypy
🔇 Additional comments (4)
plugins/aws/example/aws_realtime_function_calling_example.py (3)
45-68: LGTM: Safe mock implementation for demo purposes.The weather function returns hardcoded data with a sensible fallback. This approach is appropriate for an example file demonstrating the function-calling mechanism.
74-104: LGTM: Secure arithmetic with explicit operators.The calculation function correctly uses an explicit operator dictionary rather than dynamic evaluation, and properly handles division by zero and invalid operations. This is the safe pattern for calculator functions.
126-127: LGTM: Standard async entry point.plugins/aws/example/aws_llm_function_calling_example.py (1)
87-88: LGTM: Standard async entry point.
- Reduce connection setup logs from 7+ to 2 INFO messages - Change most event handling logs to DEBUG level instead of INFO - Remove duplicate 'Response processing error' log message - Remove unused _pending_tool_uses dictionary - Remove redundant debug logs in send_raw_event
- Add type annotation for tool_calls list - Use cast() to properly type NormalizedToolCallItem lists for _dedup_and_execute - Import cast from typing module
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
plugins/aws/vision_agents/plugins/aws/aws_realtime.py (1)
399-409: Incomplete JSON escaping can produce malformed events.The manual escaping only handles
",\n, and\r, missing backslashes (\), tabs (\t), form feeds (\f), and other control characters. This can result in invalid JSON being sent to AWS Nova when content contains these characters.Use
json.dumps()for proper escaping:async def text_input(self, content_name: str, content: str): - # Escape content for JSON - escaped_content = ( - content.replace('"', '\\"').replace("\n", "\\n").replace("\r", "\\r") - ) - event_json = self.TEXT_INPUT_EVENT % ( - self.session_id, - content_name, - escaped_content, - ) + event = { + "event": { + "textInput": { + "promptName": self.session_id, + "contentName": content_name, + "content": content, + } + } + } + event_json = json.dumps(event) await self.send_raw_event(event_json)
🧹 Nitpick comments (6)
plugins/aws/vision_agents/plugins/aws/aws_realtime.py (6)
103-136: Consider using json.dumps for template construction.The string-based event templates with
%splaceholders are prone to injection issues if any interpolated values contain special JSON characters. While this matches AWS samples, constructing events via dictionaries andjson.dumps()would be more robust and maintainable.Apply this pattern:
- TEXT_CONTENT_START_EVENT = """{ - "event": { - "contentStart": { - "promptName": "%s", - "contentName": "%s", - "type": "TEXT", - "role": "%s", - "interactive": false, - "textInputConfiguration": { - "mediaType": "text/plain" - } - } - } - }""" + @staticmethod + def _build_text_content_start_event(prompt_name: str, content_name: str, role: str) -> str: + return json.dumps({ + "event": { + "contentStart": { + "promptName": prompt_name, + "contentName": content_name, + "type": "TEXT", + "role": role, + "interactive": False, + "textInputConfiguration": { + "mediaType": "text/plain" + } + } + } + })
219-223: Raise a specific exception type.Rather than raising a generic
Exception, define or use a more specific exception class (e.g.,ConnectionErroror a customBedrockConnectionError). This allows callers to handle connection-specific failures more precisely.- raise Exception( - "AWS Bedrock requires system instructions before sending regular user input" - ) + raise ConnectionError( + "AWS Bedrock requires system instructions before sending regular user input" + )
319-333: Consider extracting inference configuration as parameters.The hardcoded
maxTokens: 1024,topP: 0.9, andtemperature: 0.7values in the session start event should ideally be configurable via__init__parameters to allow users to customize inference behavior.
541-606: LGTM with minor refactor suggestion.The tool call handler correctly extracts inputs, executes the tool with timeout, and sends results back to Nova. The nested try/except for error response sending ensures resilience even when error reporting fails.
The nested try/except could be slightly clearer by extracting the error sending logic:
async def _send_tool_error(self, tool_use_id: str, error_msg: str, tool_name: str): """Send tool error result back to Nova.""" try: content_name = str(uuid.uuid4()) await self.send_tool_content_start(content_name, tool_use_id) await self.send_tool_result(content_name, {"error": error_msg}) await self.content_end(content_name) except Exception as e: logger.error(f"Failed to send error response for tool {tool_name}: {e}", exc_info=True)
745-757: Track spawned tool execution tasks.The
asyncio.create_task()call spawns tool execution without storing the task reference. If the connection closes or the event loop shuts down before tool execution completes, errors may be lost and cleanup may not occur. Consider tracking tasks in a set and awaiting them during shutdown.def __init__(self, ...): ... + self._tool_tasks: Set[asyncio.Task] = set() async def _handle_events(self): ... - asyncio.create_task( + task = asyncio.create_task( self._handle_tool_call( tool_name=tool_call_info["toolName"], tool_use_id=tool_call_info["toolUseId"], tool_use_content=tool_call_info["toolUseContent"], ) ) + self._tool_tasks.add(task) + task.add_done_callback(self._tool_tasks.discard)And await pending tasks in
close():async def close(self): ... # Wait for pending tool calls if self._tool_tasks: await asyncio.gather(*self._tool_tasks, return_exceptions=True) ...
784-792: Consider adding circuit breaker for repeated errors.The exception handler logs errors and continues processing, which provides resilience. However, if errors occur repeatedly (e.g., persistent validation issues), the loop could continue indefinitely logging the same error. Consider tracking consecutive errors and breaking after a threshold.
consecutive_errors = 0 MAX_CONSECUTIVE_ERRORS = 10 while True: try: ... consecutive_errors = 0 # Reset on success except StopAsyncIteration: ... except Exception as e: consecutive_errors += 1 logger.error(f"Error in event handling ({consecutive_errors}/{MAX_CONSECUTIVE_ERRORS}): {e}") if consecutive_errors >= MAX_CONSECUTIVE_ERRORS: logger.error("Too many consecutive errors, stopping event loop") break continue
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
plugins/aws/vision_agents/plugins/aws/aws_realtime.py(12 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (.cursor/rules/python.mdc)
**/*.py: Do not modify sys.path in Python code
Docstrings must follow the Google style guide
Files:
plugins/aws/vision_agents/plugins/aws/aws_realtime.py
🧠 Learnings (1)
📚 Learning: 2025-10-13T22:00:34.300Z
Learnt from: dangusev
Repo: GetStream/Vision-Agents PR: 98
File: plugins/deepgram/vision_agents/plugins/deepgram/stt.py:135-150
Timestamp: 2025-10-13T22:00:34.300Z
Learning: In the Deepgram STT plugin (plugins/deepgram/vision_agents/plugins/deepgram/stt.py), the `started()` method is designed to wait for the connection attempt to complete, not to guarantee a successful connection. It's acceptable for the connection attempt to fail, and downstream code handles the case where `self.dg_connection` is `None`. The `_connected_once` event is set in the `finally` block intentionally to signal attempt completion.
Applied to files:
plugins/aws/vision_agents/plugins/aws/aws_realtime.py
🧬 Code graph analysis (1)
plugins/aws/vision_agents/plugins/aws/aws_realtime.py (3)
agents-core/vision_agents/core/llm/realtime.py (2)
Realtime(24-203)_emit_audio_output_event(119-131)agents-core/vision_agents/core/llm/llm.py (3)
_build_enhanced_instructions(61-85)get_available_functions(185-187)_run_one_tool(231-297)plugins/aws/vision_agents/plugins/aws/aws_llm.py (1)
_convert_tools_to_provider_format(625-674)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: unit / Ruff & mypy
- GitHub Check: unit / Test "not integration"
- GitHub Check: unit / Test "not integration"
- GitHub Check: unit / Ruff & mypy
🔇 Additional comments (11)
plugins/aws/vision_agents/plugins/aws/aws_realtime.py (11)
1-25: LGTM!Imports are clean and properly organized. The
PcmDataimport aligns with the audio handling requirements for the Realtime implementation.
38-98: LGTM!The expanded docstring effectively documents the new function calling capabilities with clear examples. The documentation follows Google style guidelines and provides helpful reference links to AWS Nova samples.
138-176: LGTM!The initialization properly configures the audio track for Nova's 24kHz output and adds appropriate state tracking for pending tool calls. The
_pending_tool_callsdictionary structure is well-suited for managing tool lifecycle.
233-255: LGTM!The audio response handling correctly validates connection state and resamples to Nova's required 24kHz format before processing.
274-283: LGTM!The content input wrapper correctly sequences the three required events with appropriate debug logging.
391-397: LGTM with caveat.The method correctly formats the content start event. However, if
rolevalues ever include special characters, the string interpolation could produce invalid JSON. This is low risk given typical role values ("USER", "SYSTEM", "TOOL"), but worth noting.
411-413: LGTM!The content end event is correctly formatted with controlled values.
415-424: Verify that silently swallowing all send errors is safe.The broad exception handler logs but doesn't raise, which could mask critical issues like serialization failures or credential problems. While resilience to transient network errors is valuable, silently ignoring all exceptions might hide bugs or configuration issues during development.
Consider raising for specific error types that indicate programmer error:
except json.JSONDecodeError as e: logger.error(f"Invalid event data - cannot serialize to JSON: {e}") raise # This is a programming error, not a transient failure except Exception as e: logger.error(f"Failed to send event to AWS Nova: {e}") # Don't raise for network errors to prevent connection reset
491-514: LGTM!The tool content start event is correctly structured with proper role and configuration fields for Nova's tool result flow. Docstring follows Google style guidelines.
516-539: LGTM!The tool result method correctly stringifies results as required by Nova's API. The handling of both string and non-string results is appropriate.
794-803: LGTM!The finally block appropriately logs connection state to aid debugging, helping distinguish between normal shutdowns and unexpected disconnections.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
plugins/aws/vision_agents/plugins/aws/aws_llm.py (1)
643-662: Consider handling non-object schema types.The JSON Schema normalization at lines 647-662 assumes tool parameters are objects, which is reasonable for most cases. However, if
paramsalready has a"type"field with a non-object value (e.g.,"string"or"array"), line 662 still adds"additionalProperties": False, which is only valid for object types.This is likely a non-issue in practice since tool parameters should be objects, but for robustness you could add a check:
else: # Already has type, but ensure additionalProperties is set for objects if params.get("type") == "object" and "additionalProperties" not in params: params["additionalProperties"] = False
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
plugins/aws/vision_agents/plugins/aws/aws_llm.py(15 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (.cursor/rules/python.mdc)
**/*.py: Do not modify sys.path in Python code
Docstrings must follow the Google style guide
Files:
plugins/aws/vision_agents/plugins/aws/aws_llm.py
🧬 Code graph analysis (1)
plugins/aws/vision_agents/plugins/aws/aws_llm.py (6)
plugins/aws/tests/test_aws.py (1)
llm(37-44)agents-core/vision_agents/core/llm/events.py (2)
LLMResponseChunkEvent(90-105)LLMResponseCompletedEvent(109-115)plugins/aws/vision_agents/plugins/aws/aws_realtime.py (1)
_convert_tools_to_provider_format(437-489)agents-core/vision_agents/core/llm/llm.py (2)
LLMResponseEvent(25-29)_dedup_and_execute(318-350)agents-core/vision_agents/core/llm/llm_types.py (2)
NormalizedToolCallItem(107-111)ToolSchema(64-67)plugins/aws/vision_agents/plugins/aws/events.py (1)
AWSStreamEvent(7-10)
🔇 Additional comments (21)
plugins/aws/vision_agents/plugins/aws/aws_llm.py (21)
1-81: LGTM on imports and initialization.The addition of logging, event imports, and the
_pending_tool_uses_by_indexdictionary for tracking streaming tool uses are appropriate for the enhanced functionality.
82-102: LGTM on simple_response method.Docstring formatting improvements with no functional changes.
113-121: LGTM on tool configuration setup.The tool conversion and wrapping in
toolConfigis correct, and system instructions are properly captured for use in follow-up calls.
124-131: LGTM on conversation history handling.Proper defensive checks and message normalization ensure conversation context is preserved correctly.
173-195: LGTM on tool result formatting.The conversion of tool results to text format and proper error handling ensures compatibility with AWS Bedrock's expected
toolResultstructure.
197-245: LGTM on follow-up call handling.The multi-round tool execution logic correctly preserves system parameters, handles errors comprehensively, and exits the loop when a final text response without tool calls is received.
247-273: LGTM on final pass handling.Correctly omits
toolConfigin the final call while preserving system parameters, with consistent error handling.
275-289: LGTM on final event emission.Proper validation and warning for empty responses, followed by consistent event emission.
291-314: LGTM on error handling.Comprehensive exception handling that extracts AWS error details, logs with stack traces, and preserves exceptions for upstream handling.
317-345: LGTM on streaming method setup.The streaming path correctly mirrors the non-streaming setup for tools, conversation history, and system instructions.
347-370: LGTM on initial streaming call.Proper error handling and validation of the stream field in the response.
372-401: LGTM on stream processing and assistant message construction.Correctly processes streaming events and constructs the assistant message with
toolUseblocks in AWS Bedrock format.
404-406: Same type cast issue as in non-streaming path.The cast at line 405 has the same type consistency concern as mentioned for the non-streaming path (line 164). Consider updating
_dedup_and_executeto acceptList[NormalizedToolCallItem]for better type safety.
433-480: LGTM on streaming follow-up rounds.The streaming follow-up logic correctly mirrors the non-streaming path, preserving system parameters and accumulating text across rounds.
483-500: LGTM on final streaming pass.Consistent with the non-streaming final pass logic, correctly omitting
toolConfigwhile preserving system parameters.
502-508: LGTM on streaming completion.Proper aggregation of text parts and consistent event emission.
516-579: LGTM on stream event processing.The event processing logic correctly handles text deltas and tool use blocks, properly accumulating tool inputs across streaming events.
581-623: LGTM on extraction helper methods.Both methods include proper defensive checks and correctly extract data from AWS Bedrock response structures.
676-706: LGTM on message normalization.The method properly handles various input formats and extracts text content from AWS message structures.
163-165: No action needed—the cast is appropriate by design.The
_dedup_and_executemethod intentionally accepts genericList[Dict[str, Any]]rather thanList[NormalizedToolCallItem]. SinceNormalizedToolCallItemis aTypedDict, the cast is both valid and necessary to satisfy the function signature. This pattern is consistent across the codebase—other implementations (gemini, anthropic) suppress the same type check using# type: ignore[arg-type]. The explicitcast()in aws_llm.py is actually more transparent than the type ignore approach. No type safety issue exists here.Likely an incorrect or invalid review comment.
625-674: This review comment is incorrect and should be disregarded.The Converse API (boto3 SDK) expects
inputSchema.jsonto be a native object/dict, while different APIs may have different requirements. The code is correct:
aws_llm.py (line 669): Uses the Bedrock Converse API via
boto3.client("bedrock-runtime")and correctly passesparamsas a dict, matching the boto3 Converse client requirement to pass a native dict for inputSchema.json.aws_realtime.py (line 484): Uses the separate Bedrock Realtime Streaming API with a different client (
BedrockRuntimeClient) and correctly stringifies the schema. These are distinct APIs with different requirements.The review conflates two separate AWS Bedrock APIs. The format differences are intentional and correct for each respective API, not an inconsistency requiring correction.
Likely an incorrect or invalid review comment.
Note
Adds AWS Bedrock Realtime (Nova Sonic) function calling with tool schema conversion and execution handlers, plus core agent/MCP updates, examples, and tests.
plugins/aws/vision_agents/plugins/aws/{aws_realtime.py,aws_llm.py}.plugins/aws/example/*and tests inplugins/aws/tests/test_aws_realtime.py.vision_agents/core/agents/agents.py,events/manager.py,processors/base_processor.pyto support tool-calling workflows and realtime/streaming events.core/mcp/{mcp_base.py,mcp_manager.py}for tool execution routing.core/utils/logging.py.plugins/{cartesia,elevenlabs,kokoro}/vision_agents/plugins/*/tts.pyandplugins/getstream/vision_agents/plugins/getstream/stream_edge_transport.py.plugins/krisp/vision_agents/plugins/krisp/turn_detection.py.examples/01_simple_agent_example/*) and lock/config files (pyproject.toml,uv.lock).Written by Cursor Bugbot for commit 5914a7e. This will update automatically on new commits. Configure here.
Summary by CodeRabbit
New Features
Documentation
Tests
Chores